Video surveillance is critical in many circumstances. One problem with video surveillance is that videos are manually intensive to monitor. Video monitoring can be automated using intelligent video surveillance systems. Based on user defined rules or policies, intelligent video surveillance systems can automatically identify potential threats by detecting, tracking, and analyzing targets in a scene. However, these systems do not remember past targets, especially when the targets appear to act normally. Thus, such systems cannot detect threats that can only be inferred. For example, a facility may use multiple surveillance cameras to that automatically provide an alert after identifying a suspicious target. The alert may be issued when the cameras identify some target (e.g., a human, bicycle, or vehicle) loitering around the building for more than fifteen minutes. However, such system may not issue an alert when a target approaches the site several times in a day.